🧠

AI ANN Neural Networks - Zero to Hero Andrej Karpathy

The spelled-out intro to neural networks and backpropagation: building micrograd

Resources

Notes

  • Step by step instructions for building micrograd
  • "micrograd is all you need to train a artificial neural network, everything else is efficiency"
  • The backpropagation part is only ~100 lines of Python
  • What is a derivative measuring?
    • If you increase some number ($a$) by some small number ($h$) with what sensitivity does the slope respond?
    • Does the function go up or down? By how much?
  • Eventually you get a lot of connected neurons and a loss function
    • The loss measures the accuracy of the neural net
    • We backpropagate with respect to the accuracy, trying to increase it
  • Tensors are n-dimensional arrays of scalars
  • The tuning of the loss function is a subtle art
    • Too low of a learning rate --> too long to converge
    • Too high of a learning rate --> unstable, loss could explode
  • It's common to forget to zero out the grads prior to performing backpropagation

Backlinks

Neural Networks - Zero to Hero